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About Our Working Papers 


Transforming Education is pleased to issue a series of working papers that are meant to distill information of 
value to educators, policymakers, and others in the field of Mindsets, Essential Skills, & Habits (MESH) in a 
form that can be readily updated as knowledge continues to emerge and be refined. Our working papers sum- 
marize the current state of knowledge and evidence about which skills matter for success in school, college, 
career, and life; how we can responsibly measure and build those skills; and which supports are needed for 
districts and schools to implement best practices. Because the MESH field is constantly evolving, we expect 
to revise out working papers periodically. Moreover, we hope educators, researchers, and policymakers will 
share additional research and effective practices related to MESH skill development. 


If you have feedback on this working paper or want to share your own approach to incorporating MESH in 
your district or school, please e-mail press@transformingeducation.org, 
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We have known for a long time that academic performance is one of many 
factors that make a great school, but CORE districts are now serving as 

a model for how we can actually measure these factors and look more 
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help schools and districts meet the needs of our students. 


RAMON CORTINES 
Former Superintendent 
Los Angeles Unified School District 


It’s a flashlight, not a hammer. The school districts participating in the 
development and use of the Index have chosen to shine a brighter, wider 
light on the needs of all students and on their own educational strategies and 
practices. They are not seeking to avoid accountability, but rather to create a 
better, more comprehensive system to inform and guide the efforts of schools 
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MICHAEL HANSON 
Superintendent 
Fresno Unified School District 


[The CORE Districts' multi-metric Index]... provides actionable places for 
school leaders and communities to focus their improvement work... The 
Index helps us to learn from schools that are succeeding, as well as to better 
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among schools that may be in need of intervention and support.? 


RICHARD CARRANZA 


Former Superintendent 
San Francisco Unified School District 
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Introduction 


States across the country are beginning to consider the opportunities that the Every Student Succeeds Act 
(ESSA) presents to expand their definition of student success. As state leaders engage a diverse group of 
stakeholders and consider how to act on research about the broad range of factors that prepare students for 
college, career, and life, the CORE Districts offer one example of how the new flexibility of ESSA can sup- 
port a more holistic vision of student success and school quality. 


The CORE Districts—trepresenting Los Angeles, Long Beach, Santa Ana, Garden Grove, Fresno, San Fran- 
cisco, Oakland and Sacramento unified districts—have a six-year history of collaboration to innovate, im- 
plement, and scale strategies that improve outcomes for over one million students. Through a No Child Left 
Behind waiver granted in 2013, the CORE Districts have implemented a holistic system of accountability and 
continuous improvement that focuses on students’ social-emotional skills and school climate/culture along- 
side academic outcomes.* A deep commitment to peer learning and mutual accountability amongst educators 
serves as the bedrock of the system, reflecting the district leaders’ shared belief that accountability should be 
linked to meaningful capacity building and motivated by educators’ intrinsic desire to help students succeed. 


This case study provides an overview of the CORE Districts’ groundbreaking data system and the ways in 
which this system has been operationalized in practice. We will look in depth at how social-emotional com- 
petencies—a key component of the system—were prioritized and assessed. We will conclude with lessons 
learned about the CORE Districts’ innovative system that can inform other next-generation assessment and 
continuous improvement efforts, including those catalyzed by ESSA. 


Embedded throughout the case study are links to select tools that states and districts may use to learn from, 
adapt, or replicate components of the CORE Districts’ approach. In providing these simple tools, we seek to 
provide insight into individual pieces of the complex, multi-year undertaking of the CORE Districts and to 
highlight key elements that may translate to other education systems. It is important to note that the CORE 
Districts were able to take on this ambitious body of work in part because they partnered with researchers, 
funders, and technical assistance providers in areas ranging from assessment to continuous improvement 
methodology. With the benefit of these additional resources and through deep collaboration with educators, 
district leaders, and external partners, the CORE Districts have built a compelling example of what is possi- 
ble when we expand our definition of student success to include a broad range of factors that help students 
thrive not only in school, but also in life. 
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I. Introduction: Development of the CORE Districts’ No Child Left Behind 
Waiver and Overview of the School Quality Improvement System 


History of the CORE Districts 


The roots of the CORE Districts are grounded in the California Collaborative on District Reform—estab- 
lished in 2006 to create a network of education reformers committed to learning from each other by sharing 
best practices—and in the Urban Education Dialogue (UED), a group of urban superintendents who met 
regularly to share challenges they faced, discuss programs that were buildings solutions, and provide im- 
portant peer feedback.’ Several superintendents who participated in these groups collaborated with other 
stakeholders from across the state to design a federal Race to the Top (phase I) application for California.° 
While the application was unsuccessful, the participating superintendents decided to join together to build 
upon the principles of district collaboration to propel innovation. In 2010, ten districts went on to form the 
CORE Districts under the auspices of California Education Partners (Ed Partners), which was founded by 
Rick Miller, Phil Halperin, and Natasha Hoehn.’ The original CORE districts were Clovis, Fresno, Garden 
Grove, Long Beach, Los Angeles, Oakland, Sacramento, San Francisco, Sanger, and Santa Ana. Between 2010 
and 2014, the CORE districts collaborated on a number of joint projects such as the development of perfor- 
mance assessments and the rollout of the Common Core. Based on their shared vision of collaboration that 
improves student outcomes and governance that enables flexibility, innovation, and dissemination of best 
practices, the CORE districts decided to jointly submit a No Child Left Behind (NCLB) waiver application as 
a consortium of districts. 


Developing the No Child Left Behind Waiver 


In response to criticism about NCLB’s stringent requirements, the U.S. Department of Education (USED) 
began allowing states to submit applications to waive select provisions of NCLB by demonstrating a robust, 
state-created accountability plan to “improve educational outcomes for all students, close achievement gaps, 
increase equity, and improve the quality of instruction.’”’* As of 2015, there were 43 states that had waivers 
approved by USED. 


In 2013, Rick Miller, the CORE Districts' Executive Director, and Michelle Steagall, the CORE Districts! 
Chief Academic Officer, led the effort to develop an NCLB waiver application’ with the original CORE dis- 
tricts. The CORE Districts Board, which is comprised of the districts’ superintendents, helped make critical 
decisions about the content of the waiver. The CORE Districts wanted to pursue a vastly different type of 
accountability system that analyzed districts’ performance across a range of measures that encompassed the 
whole child, rather than solely looking at standardized test scores and graduation rates. 


In particular, the CORE Districts wanted to incorporate social-emotional (SE) skills and school culture / 
climate (CC), alongside academics in a holistic index of school quality. They prioritized SE skills because of: 
1) the research demonstrating that these skills were key to students’ academic, career, and life success;'° 2) 

the benefits that Oakland and Sacramento had seen in implementing social-emotional learning (SEL) in their 
districts; and 3) the CORE districts’ common recognition that these skills were an important element that was 
missing from their existing data dashboards given their belief in measuring what matters. 


Ultimately, the approach that the CORE Districts proposed in their waiver application was novel both in the 
metrics it used and in the underlying values it embodied. The following excerpt from the CORE Districts' 
waiver application cover letter highlights these values: 
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‘With this waiver request, the participating districts do not seek to escape from accountability. Instead, they seek a 
waiver to [create] a new system with a higher level of shared responsibility and accountability and are ready to be 
held to a more comprehensive and higher standard on a range of measures that collectively are superior indicators of 
students’ college and career readiness, and more effective drivers of change.”"? 


The CORE Districts' School Quality Improvement System 


In August 2013, USED approved the the CORE Districts' NCLB waiver application, authorizing their 
innovative School Quality Improvement System (SQIS). The key tenets of the SQIS are captured in its four 
“foundational goals:” 


1. College and career ready expectations for all students. This indicates a commitment to equity 
through high expectations for all students. 


2. A focus on collective responsibility, accountability, and action that emphasizes capacity-building 
over accountability. 


3. The development of intrinsic motivation for change through differentiated recognition, account- 
ability, and support for schools. 


4. Focused capacity-building for effective instruction and leadership." 


The first principle indicates support of high standards for all students, coupled with a strong commitment 

to eliminate disparity and disproportionality across subgroups of students. Rather than the CORE Districts 
issuing mandates on how to do this, the individual districts make their own instructional and programming 
decisions, which exemplifies the CORE Districts’ commitment to balancing system coherence with district 
autonomy. The second and third principles focus on developing a holistic measurement system (explained 
below) in which all educators assume collective responsibility for all students’ success, rather than employ- 
ing the type of punitive and compliance-oriented accountability that operates in so many education systems. 
Pursuant to the fourth principle, capacity building for schools and districts occurs by pairing higher- and low- 
et-performing schools in school pairings or communities of practice (see pp. 21-22 for more detail) to engage 
in continuous improvement. Moreover, the CORE Districts offer professional development, tools, research, 
convenings to support educators and district leaders in sharing best practices and lessons learned. 


School Quality Improvement Index 


The CORE Districts’ School Quality Improvement Index (“SQU” or “Index”’) is a key component of the 
School Quality Improvement System (“SQIS”). While the SQIS refers to the full system of accountability and 
continuous improvement (including capacity building structures like the school pairings and communities of 
practice), the Index is the quantitative formula used to assess school quality. The Index includes measures of 
academic performance, social-emotional (SE) competencies, and culture/climate (CC) in a holistic measure of 
student success and school quality. 


The Index was rolled out in several phases over the course of 3 years (2013-16; see Appendix 2 for the full 
rollout timeline). The diagram above shows the full index once its rollout was complete. Academics account 
for 60% of the Index, while social-emotional and school culture/climate factors account for 40%.'* (See Fig- 
ure 1 and Box III for a detailed description of each component of the Index.) 


The decisions that the districts made regarding weighting allocations and methods for calculating particu- 
lar indicators reflect the core values undergirding their system. For example, to prioritize the elimination of 
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FIGURE 1 The CORE Districts' accountability system incorporates academic, SEL, 
and culture/climate measures 
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disparity and disproportionality between subgroups,'* the CORE Districts decreased the “n size,” which is the 
minimum number of students in a subgroup for which they will report data. Instead of 100 students—the 

“n size” that many school districts use—the CORE districts report data on all subgroups with 20 or more 
students. This ensures that a greater number of students are included in the conversation when the CORE 
districts reflect on student outcomes, while also balancing concern for student privacy. Similarly, to prioritize 
their commitment to continuous improvement, the CORE Districts have committed to revising the weighting 
of different variables over time based on feedback from participating stakeholders and new developments in 
research. 
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Selecting Social-Emotional Competencies to Measure 


While the CORE Districts committed in their waiver to measure students’ social-emotional development, 
they did not identify the specific social-emotional competencies or assessments they would use. Accordingly, 
one of the first steps after the waiver’s approval was to prioritize an initial set of social-emotional competen- 
cies and identify promising measures for each. In November 2013, the CORE Districts convened social-emo- 
tional learning (SEL) experts and representatives from each of the CORE districts, including superintendents, 
directors of student support, directors of social-emotional learning, and directors of special education. The 
SEL experts in attendance were from the Collaborative for Academic, Social and Emotional Learning (CA- 
SEL), the John W. Gardner Center for Youth at Stanford, and TransformEd. TransformEd recommended 
selecting specific social-emotional competencies based on the extent to which each competency was mean- 
ingful, measureable, and malleable (see Box I below for more information on the “3Ms”). Additionally, the 
CORE Districts prioritized identifying at least one intrapersonal skill and one interpersonal skill in their initial 
set of SE competencies. This ensured that there was a broad array of competencies that would yield comple- 
mentary data. 


BOX I TransformEd’s 3's Filter 


TransformEd proposed three key criteria for selecting social-emotional competencies: they needed to be 
meaningful, measureable, and malleable (or meet the “3M’s test”).'° 


MEANINGFUL indicates that a particular competency is predictive of important academic, career, and life outcomes. 


MEASUREABLE indicates that the competency can be measured reliably through a valid assessment that 
waist syle) (om colr-\elaalialisi\clmr-imcier| (= Mla esealele) ce 


MALLEABLE indicates that there is research showing that the competency can be developed in a school setting. 


Click here to 


download the tool. 


Ultimately, district representatives and SEL content experts used a voting process to prioritize specific SE 
and CC competencies for inclusion in the SQU. Through this process, four SE competencies were prior- 
itized: growth mindset, self-efficacy, self-management, and social awareness. (See Box II on the next page 

for definitions of the competencies.) The CORE Districts acknowledge that this is not a comprehensive set 
of SE competencies; rather they believe that it is a reasonable starting point that passes the “3Ms Test” and 
includes a mix of intrapersonal and interpersonal skills. The CORE Districts may choose to refine this set of 
skills over time as further research emerges. For example, collaborative problem solving was almost incorpo- 
rated into the initial set of competencies, but the CORE Districts elected to wait until the PISA 2015 test had 
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piloted new performance-based measures of this competency before considering it for inclusion in the SQU 
in future years. 
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BOX II Definitions of the Four Social-Emotional Competencies Prioritized 
by the CORE Districts 


GROWTH MINDSET: The belief that one’s abilities can grow with effort. Students with a growth 
mindset see effort as necessary for success, embrace challenges, learn from criticism, and persist in the face 
of setbacks." 


SELF-EFFICACY: The belief in one’s own ability to succeed in achieving an outcome or reaching a 


goal. Self-efficacy reflects confidence in the ability to exert control over one’s motivation, behavior, and 
Yanviine)alant=0) ela 


SELF-MANAGEMENT: The ability to regulate one’s emotions, thoughts, and behaviors effectively 
Tame lix=1x=\almesyi(l-1t(e)atseum Mal ism (are [0(e{-tom par-larcleliale Moiinextomme(=)i- Na lale mel e-iiji(eroli(e)ammanlelhc-lilale Me)al-\sX-)|mar= Tale McX>14il ale} 
and working toward personal and academic goals.'° 


SOCIAL AWARENESS: The ability to take the perspective of and empathize with others from 
diverse backgrounds and cultures, to understand social and ethical norms for behavior, and to recognize 
family, school, and community resources and supports.”° 


While the process of selecting competencies included stakeholders from within each district, the primary 
decision-makers were SEL content experts and district-level staff. Unfortunately, the CORE Districts were 
unable to conduct a more comprehensive stakeholder engagement process for the competency selection com- 
ponent of the waiver due to the time constraints in place: the CORE Districts received approval in August 
2013 and had to administer social-emotional surveys just a few months later in order to pilot the measures 
before rolling them out district-wide. That said, the initial development of the waiver—including the idea of 
measuring SE skills as part of the SQU—incorporated feedback from key stakeholder groups such as teachers, 
principals, community members, and parents. The CORE Districts have continued to engage these stake- 
holders throughout the implementation of the waiver. If other districts or states undergo a similar process, 
they should consider building in additional time to incorporate ample input from teachers, principals, parents, 
community members, and employers into the decision-making process (e.g., through surveys, focus groups, 
and community meetings, etc.). Doing so builds broader buy-in and ensures that the competencies selected 
are responsive to community needs and values. 
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Overview of All Index Indicators 


The four SE competencies described above make up one part of the holistic SQU. The other indicators 
included in the SQU are described in more detail in Box III. 
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BOX III Description of All Index Components 


Social-emotional & culture/climate Academics 


CHRONIC ABSENTEEISM: The percent of students 
who have an attendance rate at or below 90% 
within a given school year. 


CULTURE/CLIMATE SURVEYS: Results from surveys of 
students (grades 4-12), teachers, and parents that 
Tare [Ule{-me[0(-t=14(e)atswr=] ofeljar- Wer llaar=](oMelmc10] 0) ele)amie)e 
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connectedness; knowledge and perceived fairness 
of discipline rules and norms; and safety. 


SUSPENSION/EXPULSION RATES: The percent of 
students who are suspended and/or expelled at 
least once in a given school year. 


SOCIAL-EMOTIONAL SKILLS: Student self-report sur- 
veys in grades 4-12 that measure growth mindset, 
self-efficacy, self-management, and social aware- 
ness. (Some districts also use teacher reports on 
students’ SE competencies, though these do not 
currently factor into the districts’ Index score.) 


ELL RE-DESIGNATION RATES: This measure captures 
the percentage of students who move from English 
language learner status to “fluent English proficient 
status, rather than becoming “long-term English 
learners.” 


” 


Special education identification (information only) 
Identifies subgroups that are overrepresented in 
special education (when the overrepresentation is 
Statistically significant with 99% confidence). 


PERFORMANCE: The percent of students at grade 
level for English/language arts and mathematics, 
based on Smarter Balanced Assessment 
Consortium (SBAC) test scores" 


GROWTH:” For the purposes of the Index, the 
CORE Districts Growth model will be designed to 
look at the extent to which schools have helped 
FSi (Uyol=Yalccmanre) om ike)tam Oe) /n @AWCoM ofe) [alm = Ma-\l- theme) 
students who started the school year in a similar 
place (e.g., in terms of prior achievement and 
potentially in terms of observable demographics 
like English Learner status or free and reduced 
price lunch status). For more details on how the 
CORE Districts plan to measure growth, please see 
i (al=y (am (0) | Manloieuleme(=Vilaliteyan 


HIGH SCHOOL READINESS:” The percent of all 8th 
grade students who meet the following criteria: 1) 
8th grade GPA of 2.5 or higher; 2) attendance of 
96% or higher; 3) no final course grades of D or F 
in ELA or math; and 4) were not suspended in 8th 
grade.” 


GRADUATION: The 4-year cohort graduation rate, 
N/a l(e;amere)at=j(o(=1e-m ale\wanaelanvaesiiele(=\altome) r= le L0r=1() 
compared with the number of students enrolled in 
the school (accounting for students who transfer 
Talcode= Tale Mel0 lao) im tal-msie alele)) Mrorar-|aleMomaV{-r-lmerelale)at 
graduation rates are also included in the Index 
results. 


ll. Moving from Competencies to Measures 


This section explains the process the CORE Districts used to select, pilot and field test measures of the SE 
and CC competencies prioritized for inclusion in the Index. 


Selecting the Social-Emotional Measures 


Once the SE competencies of growth mindset, self-efficacy, self-management, and social awareness were 
prioritized (see pp. 8), TransformEd conducted a scan of the field to identify the most promising measures 
for each competency, using the following set of criteria: 
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Evidence-based: The measures needed to demonstrate emerging evidence of validity and reliability 
(further evidence would be gathered through the CORE Districts’ own pilot and field test). They 
must also have “face validity” for educators, meaning that educators believe the scales measure what 
they intend to measure. 


Free to administer: To ensure the financial sustainability of the CORE Districts' system, the measures 
needed to be free to use. In order to secure free measures, TransformEd went directly to the leading 
SEL researchers to obtain permission to use the SE measures they had developed. (See Figure 2 on 
next page outlining each competency and the associated researcher’s measures.) While this process 
was time-consuming, it was preferable to going through third party assessment providers that would 
charge a per student or per school licensing fee for the measures every year. 


Practical to administer: The measures needed to be simple to administer to students in grades 4 and 
up, with as little administrative burden as possible. Districts asked that there be both online and 
paper-based options for administering the measures given the limitations of districts’ technology 
infrastructure at the time. Further, the districts asked that the measures be feasible to administer in 
conjunction with existing survey administration efforts to minimize the burden of administering a 
separate assessment. 


Parsimonious: The assessments needed to use the fewest number of items possible to get valid, 
reliable results. This principle was responsive to concerns about “over testing” students, and to 
teachers’ and administrators’ desire to protect instructional time. The CORE Districts aimed for 
students to be able to complete SE survey items in approximately 10 minutes and CC items in 10 
minutes, which made it possible for students to complete the full survey within one class period. 
There was a clear recognition that using proxies like attendance, grades, and discipline rates could 
round out the picture of students’ SE skills without overburdening students through lengthy 
assessments. 


Strengths-based: The CORE Districts had a preference for questions framed in a positive manner 
whenever possible (unless research showed that particular negatively-phrased scale was more valid 
or teliable). An example of a positively-framed item is: “I can earn an A in my classes.” Whereas, 
the following is an example of a negatively-framed item: “My intelligence is something that I can’t 
change very much.” 


Given these criteria, TransformEd spoke with leading experts in SE assessment—including Lisa Blackwell, 
Clancy Blair, Eduardo Bricefio, Celene Domitrovich, Angela Duckworth, Carol Dweck, Camille Farrington, 
Greg Walton, Roger Weissberg, and David Yeager—and curated the best available measures based on the 


criteria outlined above. (See Figure 2 for the primary source of each measure.) 


Once TransformEd had identified potential measures based on the input of multiple researchers, district 
staff members vetted the measures and provided feedback. Ultimately, the CORE Districts Board approved 
the final set of measures in December 2013, requesting that the measures be piloted with a small group of 
schools in spring 2014 and field tested with all schools in spring 2015 before being formally included in the 
SQII. 
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TOOL V Panorama Education’s 
_ Assessment Design Checklist 
Using a research-backed approach, this tool can help you 


A NS an existing Suey design instrument or select a new 
ed © \<. Click here to download the tool. 


FIGURE 2 Key Experts Consulted 
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Pilot Testing 


The CORE Districts conducted a pilot test of the SE measures in spring 2014, collecting approximately 9,000 
student self-reports and 1,000 teacher reports. During this pilot, two different forms of the student self-re- 
port surveys and teacher surveys were randomly assigned to participants. For each competency, one of the 
forms used the original measure developed by a contributing researcher and the other form provided a mod- 
ified version developed in partnership with Dr. Hunter Gehlbach to reflect emerging best practices in survey 
design (e.g, removing double-barreled items, translating statements into questions, etc.). 


The CORE Districts, TransformEd, and researchers at Harvard’s Center for Education Policy Research 
(CEPR) compared the two forms of each measure to identify the more promising form based on the follow- 
ing criteria: 1) validity, including correlations with validating scales and correlations with external variables; 
and 2) internal reliability. The scales that performed the best across these two criteria were then tested at 
much greater scale during the 2015 field test. 
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BOX IV Mitigating the risk of biases inherent in surveys 


Prior to implementing the pilot test, the CORE Districts were concerned about addressing three major issues 
regarding bias in survey administration: reference bias, social desirability bias, and stereotype threat. The CORE 
Districts were thoughtful in implementing a variety of approaches to mitigate these concerns. Current data 
analyses show no clear evidence that these issues impacted the results of the survey.*° 


REFERENCE BIAS refers to the tendency for individuals’ survey responses to be influenced by the context in 
which the survey is administered. Experts in reference bias point out that the culture of a school might influence 
a students’ frame of reference, which may in turn cause the student to interpret a survey scale differently 

than students from other schools.”° For example, students attending a school that heavily emphasizes self- 
management might develop higher internal standards for self-management. Such students might then rate 
themselves lower on a self-reported measure of self-management than students in a school that does not 
emphasize this competency.?’ 


To mitigate concerns about reference bias, the CORE Districts partnered with ETS to pilot anchoring vignettes, 
a technique that uses brief descriptions of sample students who exhibit varying levels of the target competency 
as a tool for norming students’ responses to ensure comparability. Ultimately, the analyses conducted by 
Harvard CEPR showed that using anchoring vignettes did not improve the quality of the survey data. One 
possible interpretation of this finding is that the school cultures vis-a-vis the relevant SE competencies were 
not dramatically different from one another, so students taking the surveys at different schools had relatively 
similar frames of reference. This interpretation is supported by additional findings from Professor Marty West 
and the Harvard CEPR team. Ultimately, the CORE Districts decided not to use anchoring vignettes during the 


2014-15 field test because the participating districts felt that any potential research benefit to including them was 
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SOCIAL DESIRABILITY BIAS refers to the tendency for individuals’ survey responses to be influenced by social 
pressures.”° For example, if a survey asks how often a student is polite to adults, the student may answer “almost 
all the time,” even if the response of “almost never” more accurately reflects her behavior because she knows 
that it is socially desirable to be polite to adults. The CORE Districts' attempt to mitigate this issue in two ways: 1) 
by explicitly stating that students’ survey responses will remain confidential and will not influence grades or other 
assessments of the students’ performance; and 2) by asking the adults who proctor survey administration to stand 
at the back of the classroom instead of circulating around the classroom so that students feel they can complete 
their survey without being judged by a teacher or other adult. 


STEREOTYPE THREAT refers to the tendency for individuals’ survey responses to be influenced by the respondents’ 
perception of how people in their group (e.g., racial, ethnic, or socio-economic class) are believed by others to 
perform in that competency area.*° According to the research on stereotype threat, students who are asked to 
report their gender or ethnicity before completing an assessment are more likely to perform in a manner consistent 
with their perception of how people in their particular group are believed by others to perform. For example, 

when female students were asked to report their gender before taking a math test, they performed worse on 

that test than their peers with similar levels of math skill.2° The CORE Districts address this concern by including 
demographic questions only at the end of the survey or by removing all demographic questions from the survey 
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Pilot Test Results 


Upon completion of the pilot test, researchers at Harvard CEPR analyzed the correlation of each survey 
form with students’ GPA, suspension, absences, and standardized test scores. The scales from each form that 
demonstrated the strongest correlations with other student outcomes (e.g., grades, attendance, etc.) and met 
internal reliability standards were selected for use in the large-scale field test conducted in 2014-2015.” 


After completing the pilot study, the participating districts received reports of the SE survey data, which 
they used to inform their understanding of the connection between students’ SE skills and other student 
outcomes such as grades, test scores, and attendance. The participating districts also offered feedback to the 
CORE Districts and TransformEd about the survey administration process. For example, they suggested 
combining the social-emotional and culture-climate items into a single survey, which the CORE Districts did 
for the 2014-15 field test. 
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Field Test Results 


The CORE Districts ran a full-scale field test of the refined SE and CC student survey measures in spring 
2015 with 1,500 schools. As Figure 3 shows, student ratings on each of the measures correlate in the expected 
direction with other academic and behavioral outcomes. For example, students’ self-efficacy ratings correlate 
positively with GPA and standardized math and ELA scores (with a correlation of approximately 0.3). They 
correlate negatively with the number of days a student was suspended as well as the total number of days s/ 
he was absent (with a correlation of approximately -0.06). These correlations are all statistically significant.” 
In other words, students with stronger SE skills tend to have stronger grades and test scores, and they are less 
likely to be absent or suspended throughout the school year. 


Additionally, data analyses conducted by Harvard CEPR showed no clear evidence of reference bias. Specifi- 
cally, the relationship between students’ SE self-ratings and other student outcomes (e.g., GPA and standard- 
ized test rating) across all schools was compared to the relationship between self-ratings and other outcomes 
among students attending the same school. Finding that the relationship is consistently stronger within a 
school would suggest the existence of reference bias. On the other hand, finding that the overall relation- 
ship between SE skills and other student outcomes is stronger than the within-school relationship provides 
evidence that reference bias may not be a significant concern in this dataset. Researchers at Harvard CEPR 
found the latter: the relationship between SE self-ratings and ELA test scores across the four competencies is 
consistently stronger for the overall analysis compared to the within-school analysis. These correlations are 
all statistically significant, and similar patterns were found with GPA and mathematics test scores.” (See Box 
IV on mitigating the risk of biases inherent in surveys for more information.) 
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FIGURE 3 Correlation of Student Self-Reports (Grades 4-12) with 
Academic and Behavioral Outcomes™ 
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If the student self-report measures are accurately assessing students’ SE skills, they should also be positively 
correlated with teachers’ perceptions of those same skills. To test this, two of the CORE districts (Fresno and 
Santa Ana) chose to collect teacher ratings of students’ SE skills as a complement to student self-reports. As 
Figure 4 shows, teacher ratings of students’ SE competencies correlate in the expected direction with student 
outcomes of interest, including grades, number of courses passed, attendance, and suspensions. All of the cor- 
relations are statistically significant. These findings suggest that because students and teachers have overlapping 
but distinct perspectives of students’ skills, it may be useful to use both teacher and student reports in order to 
help triangulate students’ skills with greater accuracy. (See Measuring MESH for more information about the 
validity and reliability of both student self-reports and teacher ratings of students’ SE competencies.) 


While some CORE districts continue to use teacher ratings of students’ SE competencies in addition to stu- 
dent self-ratings, the teacher ratings are not currently included in the SQII because some districts found teacher 


FIGURE 4 Correlation of Teacher Reports with Students’ Academic 


and Behavioral Outcomes*® 
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ratings impractical to administer at scale. In some districts, teacher ratings of students’ SE competencies were 
subject to the collective bargaining process; other districts viewed rating students’ competencies as part of 
the everyday responsibilities of educators. In some cases, districts offered a stipend to teachers that partici- 
pated in a pilot of the SE rating process or enabled teachers to use existing professional development time 

to complete the surveys. All participating districts capped the number of students that each teacher rated at 
the secondary level to minimize the burden on educators. The CORE Districts continue to support interested 
districts to purse both student self-report and teacher ratings of students. One promising development is the 
integration of SE ratings with existing student report cards. 


BOX V Multiple Perspectives on the Use of Measures 


Some researchers are skeptical of the use of survey-based measures for high-stakes accountability purposes 
because they argue that the measures could be subject to reference bias, that there may be incentives to “game” 
survey responses, and that the measures may not enable adequate differentiation in schools’ performance.*° 
Nonetheless, analyses from the CORE Districts’ pilot and field test demonstrate that these measures hold promise 
as one component of a holistic and formative system like the one the CORE Districts have created. Specifically, 
the Harvard CEPR team found strong evidence that the measures were related to other outcomes that matter 
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differentiate between the performance of individual schools.°’ 


Further, it is important to note that there are several contextual factors that mitigated the CORE Districts’ concerns 
about including SE measures in their accountability and continuous improvement system. First, the CORE Districts 
use relatively low weighting for SE measures, with SE surveys accounting for just 8 points of the 100-point Index. 

Second, the “consequence” for schools that perform poorly on the overall index is that they are paired with higher- 
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accountability systems that impose punitive sanctions). And finally, the CORE Districts conducted a pilot and field 
test of the measures, collected feedback from the districts regarding implementation, and has stayed abreast of 
(of-yVi-1(e) o)anlc ial MAM Alem aator=siOlc> ante lal Mmil=)(e lame) ce {cl mcomexoyaliale(el0l<)\’mlan| cen’ {-me) ole )am alo mant=r-\c10]xo\owr-laremrelelaalialiciercliceyal 
protocols.*® 


As other states and districts consider whether and how to measure SE skills, TransformEd recommends using 

the data for formative purposes (e.g., as part of a needs assessment to support school improvement efforts) and 
engaging a variety of stakeholders in exploring other uses over time. See our ESSA policy brief for further details.°° 
There is also emerging research on next generation assessments (e.g., performance tasks and game-based 
assessments) that may mitigate some of the potential flaws of survey-based measures discussed above (see Box. 
\V for more information). As these next generation measures evolve over time, the CORE Districts may eventually 
use them to complement or replace existing survey-based SE measures. 


Selecting the School Culture/Climate Measures 


The CORE Districts embarked on a similar process to select the CC survey scales as they did with the SE 
scales. Initially, the districts participated in a prioritization exercise to select the set of CC domains for the 
surveys. There was a set of principles that guided this selection that was analogous to the principles for the 
SE process. The CC measures needed to be: evidence-based (including meaningful, measurable, and “action- 
able”); free; aligned with other surveys given by the districts; feasible to complete within 10-20 minutes; and 
applicable to students in grades 4 and above. 


The John W. Gardner Center at Stanford conducted an inventory of the CC scales the CORE districts were 
already using and a scan of the field to determine which CC measures were available to assess the CC do- 
mains that the CORE Districts prioritized (see Box VI for more information about the domains). Ultimate- 
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ly the CORE Districts' CC measures incorporated components from the California Healthy Kids Survey 
developed by WestEd, which many of the CORE districts were already administering for grant-reporting 
purposes. They also included several items on parent engagement from Los Angeles Unified School District’s 
student experience survey. The CC component of the SQU includes student, teacher, and parent surveys on 
the domains described in Box VI.” 


BOX VI Definitions of the Culture/Climate Domains 


The culture-climate survey items cover four broad domains: teaching and learning, interpersonal relationships, 
safety, and school-community engagement.“ 


TEACHING AND LEARNING 


Research indicates that a positive school climate creates an optimal environment for learning. For example, 
studies have shown that teacher practices that promote active student engagement in the learning process 
contribute to students’ academic achievement.’ 


Within this area of focus, the CORE Districts are specifically measuring climate of support for academic 
learning, defined as follows: 
Students and teachers feel that there is a climate conducive to learning and that teachers use supportive 
practices, such as encouragement and constructive feedback; varied opportunities to demonstrate knowledge 
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INTERPERSONAL RELATIONSHIPS 


Teachers’ social interactions with students directly affect students’ behavioral and emotional engagement in the 
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school and high school students. Also, when students perceive teacher-student and student-peer social supports, 
these perceptions are positively associated with self-esteem and academic grades.*? 


Within this area of focus, the CORE Districts are specifically measuring sense of belonging/school 
connectedness, defined as follows: 
A positive sense of being accepted, valued, and included by others (teacher and peers) in all school settings. 
Students and parents report feeling welcome at the school. 


SAFETY 


Feeling safe in school is positively associated with students’ academic learning and healthy youth development. 
Conversely, research indicates that schools that lack clear and supportive behavioral norms are more likely to 
experience violence, peer victimization, punitive disciplinary actions, higher levels of absenteeism, and reduced 
FeYer-Yo(Janl(em-leval(c\V/-10n(=0) 


Within this area of focus, the CORE Districts are specifically measuring: 
Sense of safety: Students and adults report feeling safe from verbal abuse, teasing, or social exclusion at 
and around school. 
Knowledge and Fairness of Discipline, Rules and Norms: The school has clearly communicated 
rules and expectations about student and adult behavior, especially regarding physical violence, verbal abuse 
or harassment, and teasing. These rules and expectations are clearly and consistently enforced. 


SCHOOL-COMMUNITY ENGAGEMENT 


Researchers at the Chicago School Research Consortium have found that schools with high relational trust (such 
as positive social relationships, both among educators and between educators, families, and community members) 
are more likely to make changes that improve student achievement. They found that positive school-community 
and school-family relationships helped to reinforce teacher capacity and promoted a safe and respectful learning 
climate.** Within this area of focus, the CORE Districts are specifically measuring school-community engagement 
through a set of survey items that families respond to. 
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While the SE measures were piloted in a small subset of schools in 2013-14, the CORE district superinten- 
dents felt that their districts were ready for a full-scale field test of the CC measures that year because they 
had been using a similar set of measures in prior years, primarily through the California Healthy Kids Survey. 
In 2014-15, the CORE Districts recommended several changes to the CC scales based on findings from the 
field test. These changes were intended to complement other Index components, remove repetitive items, 
alien with California’s Local Control and Accountability Plan (LCAP) requirements, ensure appropriate mea- 
surement of all stakeholder groups (students, staff, and parents), and improve the validity and reliability of 
the measures. Finally, the SE and CC measures were combined into a single survey in most districts to stream- 
line the administration process and minimize the amount of instructional time taken up by the surveys. 


lll. Supporting School Practice Change 

and Capacity Building 

There are a variety of mechanisms through which the CORE Districts support districts in changing school 
practice and building educators’ capacity: offering professional learning opportunities, hosting convenings, 
offering resources, providing an online platform for information-sharing, and facilitating school pairings and 
communities of practice for more intensive support to schools. The data from the SQII informs each of 
these efforts. 


Data Sources 


The CORE Districts receive a range of data through Index Reports containing all of the SQUI indicators 
and through individual school reports on SE and CC. To support the effective use of these data, the CORE 
Districts use a “Ladder of Inquiry” tool: 
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TransformEd and the CORE Districts also developed a companion guide to instructional practices that can 
support students’ SE development and cultivate a positive school culture/climate: 
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Index Reports 


The CORE Districts have developed comprehensive Index Reports for each school. Actual reports are avail- 
able at http://coredistricts.org/indexreports/ and a sample report is included below. As the sample below 
shows, Index Reports were designed to be user-friendly for those without specialized expertise in data. The 
report enables a user to view all of the SQU indicators, including three-year trends and comparisons with 
other schools and districts. The fall 2016 reports (using data from spring 2016) will include growth measures 
in ELA and math as well as data from the social-emotional and culture-climate surveys. 


FIGURE 5 Sample Index Report 
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Survey Reports 


Several districts receive detailed summaries of the SE and CC data from their survey administration part- 
ner, Panorama Education (see sample reports in Figure 6), while other districts self-administer the surveys 
and create their own reports. Districts are able to disaggregate their data by item, competency, school, and 
subgroup. This enables the CORE Districts to determine which schools may need additional supports and 
identify opportunities to eliminate disparity and disproportionality between subgroups. 
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FIGURE 6 Sample SEL Report 


Explore this topic by subgroup 


Click on a subgroup category below to see the percentage 
of favorable responses by subgroup for Growth Mindset. 
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Intensive school support and capacity building 


The CORE Districts’ SQIS was inspired by the work of Michael Fullan, whose research emphasizes the im- 
portance of educators’ intrinsic motivation to help all students succeed. As such, the SQIS focuses on provid- 
ing supports to build schools’ capacity rather than attaching punitive consequences to the Index results. All 
schools and districts receive support from the CORE Districts in the form of professional learning resources 
such as performance assessment tools for ELA and Math, and targeted peer learning opportunities related 

to school districts’ areas of interest. In addition, the CORE Districts designed a system of more intensive 
support and capacity building for “priority” schools through school pairings and for “focus” schools through 
communities of practice. (See Appendix C for more information about the CORE Districts’ criteria for iden- 
tifying priority and focus schools.) 


Communities of Practice 


Communities of practice are formed with “focus” schools that have achievement gaps, low graduation rates, 
low overall performance, or low academic performance by one or more subgroups. Communities of practice 
usually operate within a single district and serve as an opportunity for a community of educators to focus on 
a common challenge using the Plan-Do-Study-Act (PDSA) approach—a model of rapid-cycle continuous 
improvement. 


BOX VII Plan-Do-Study-Act (PDSA) Approach 


PLAN: Define the “problem of practice” or issue the 
group is attempting to address. Learn about ways 
to address the problem of practice, determine which 
intervention or strategy to test, and decide which 
data will be used to determine the efficacy of the 
intervention. 


DO: Test the intervention or strategy. 


STUDY: Collect and examine evidence about the 
efficacy of the intervention or strategy. 


ACT: Based on the data and learning, decide on the 
next steps: e.g., repeat the intervention, explore a 
different intervention, etc. 


Each “focus” school develops a two-year plan of action based on its own needs assessment and student data 
analysis. The interventions associated with the PDSA process occur three times per year. The participants in 
the communities of practice document teaching approaches, interventions attempted, and trainings provid- 
ed for teachers and staff. They consider this information alongside output from the SQII and other school 
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data to analyze which actions supported school improvement and which did not. The key learnings are then 
reported to each school’s Site Council, which considers this information in conjunction with the school 
improvement plan.*’ While the Site Council is involved in the process, receiving updates on the progress of 
each community of practice, the district is responsible for determining whether the community of practice is 
helping to improve the school’s outcomes in specific areas of need. the CORE Districts’ role is to provide the 
data, tools, and resources to inform the planning efforts as well as professional development and a learning 
community for facilitators of the communities of practice. In a survey of community of practice participants 
from 2014-2015, approximately 75% of respondents agreed or strongly agreed that the program helped their 
school improve. 
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School Pairing 


Schools that have been identified as “priority” schools due to low performance and/or low graduation rates 
ate paired with high performing “reward” schools, which share best practices and provide technical assis- 
tance.” The school pairing supports an ethic of peer accountability, rather than more traditional punitive or 
“top-down” accountability. The CORE Districts believe that pairings should be valuable to both participating 
schools, not just the lower-performing school. Data from a 2014-2015 survey of school paiting participants 
supports this notion: nearly 80% of “reward” schools and nearly 70% of “focus” or “priority” schools agreed 
or strongly agreed that participating in the program helped their school to improve.” 


School pairs engage in peer learning by focusing on one or more metrics from the SQU that align with the 
ptiority school’s needs assessment. Low-performing schools that have received funds through a School Im- 
provement Grant (SIG) continue their work to implement the school improvement plan they outlined during 
the SIG process.*! Schools without an existing school improvement plan work with their school-pairing part- 
ner to develop a plan of action using the Seven Turnaround Principles contained in the waiver application. 
The pairs may employ strategies such as school visits, joint PSDA cycles, coaching, or professional learning 
communities. Additionally, the pairs have a great deal of flexibility to use innovative approaches beyond those 
listed here, provided they develop a promising plan for school improvement. 


The CORE Districts' role in supporting school pairings have been to identify schools in each of three cat- 
egories (reward, priority, and focus) and to use the data to match schools with one another. While the im- 
provement process is primarily led by each district, the CORE Districts also provide administrative support 
and professional development through Pairing Institutes, which focus on sharing continuous improvement 
strategies, developing initial plans for the pairings, setting norms, and building relationships between schools. 
When the waiver was first approved in 2013, the CORE Districts paired schools based solely on achievement 
and graduation data because the SE and CC data had not yet been collected systematically. That said, school 
pairs were able to use the SE and CC data from the pilot and field test to inform their improvement efforts. 
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IV. Areas for Continued Exploration 


With the passage of ESSA in late 2015, the CORE Districts’ waiver expired in August 2016. The next phase 
of collective work for the districts is just beginning to emerge, and the CORE Districts opened up their 
data-sharing collaborative, inviting other districts and charter networks in California to join. By participating 
in the data-sharing collaborative, districts will have access to the full range of indicators and benchmarks 
that are included in the SQU. Participating districts will also participate in cross-district convenings twice per 
year to share lessons learned and address common challenges.” There are also a number of key steps that 
the CORE Districts have begun to explore related to the existing SQIS and the development of students’ SE 
competencies. These next steps, outlined below, may be instructive for other states or districts that choose to 
build upon the Core Districts' work. 


Evaluating the Impact of SEL Practices and the SQII Overall: 


The CORE Districts’ 1,600 schools are already employing a range of interventions, instructional approaches, 
and curricula to build students’ SE skills. For example, individual CORE districts have: 


* Provided professional development for educators using TransformEd’s growth-mindset toolkit and 
self-management toolkit, as well as a variety of other resources. 


* Created a developmentally appropriate scope and sequence for improving academic habits, motiva- 
tion, SE skills, well-being, and school CC. 


* Developed a rubric for identifying observable markers of effective instructional practices to build SE 
skills. 


* Begun mapping the intersections between SE skills and the Common Core State Standards to clarify 
how academics and SEL can be integrated seamlessly in the classroom. Added SE skills to student 
report cards to ensure that students, teachers, and parents are having regular conversations about 
students’ SE development. 
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The CORE Districts have entered into a partnership with Policy Analysis for California Education (PACE), a 
research center at Stanford that will support the districts in identifying promising practices to support stu- 
dents’ SE development. Using the common SE measures, PACE will identify “bright spots” across the 1,600 
CORE Districts! schools, or schools that demonstrate particularly strong SE outcomes for students. The 
PACE team will then conduct site visits and interviews to determine what specific practices or approaches 
may be driving these strong SE outcomes. Armed with this information and the peer learning infrastruc- 
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ture provided by the CORE Districts, the districts can work to bring the most effective approaches to scale, 
offering a range of potential options based on students’ particular SE strengths and needs. Through their 
partnership with Stanford PACE, the CORE Districts also aim to develop the relationships, data systems, 
and infrastructure to support a robust research agenda that will include evaluating the impact of the CORE 
Districts' accountability model and comparing the CORE Districts model to other accountability frameworks 
in California. 


Continuing to Engage in Cross-District Learning around SEL: 


To foster more direct connections with teachers and district staff, TransformEd and the CORE Districts are 
piloting a year-long Social-Emotional Learning Fellowship for district staff members. SEL Fellows remain 
full-time district employees while also playing a leadership role in shaping SEL-related work across the CORE 
districts. Each fellow actively gathers input from educators in his/her district to refine the CORE Districts' 
SEL-telated survey administration, data reporting, and practice improvement work. For example, Fellows 
identify needs in their own districts, help plan and facilitate CORE district-wide trainings related to SEL, and 
develop tools that can support their colleagues in other CORE districts. The CORE Districts Staff and SEL 
Fellows are also exploring whether the Networked Improvement Community (NIC) model might be an effec- 
tive way to support districts in drawing upon the SE data collected within and beyond the Index to develop, 
test, and refine interventions that improve students’ SE skills. 


Through conversations with district staff, principals, and teachers, the following opportunities for further 
cross-district learning have emerged:* 


Building Collective Ownership: Because the research is complex and nuanced, it can be challenging to 
communicate succinctly what SE skills are and why they matter. One district administrator said, “Dis- 
trict folks don’t necessarily have a firm...understanding of SEL, and it is hard for us to communi- 
cate... how transformative [SEL] is for students.” Staff members from multiple CORE districts have 
identified a need to develop more coherent messaging in order to articulate how SEL relates to other 
district priorities and to work towards a more collective sense of ownership for this work. 


Integrating SEL and Academics: District staff membets with expertise in SEL believe that SE skills 
must be fully integrated into academic content and instruction in order to meaningfully improve stu- 
dent outcomes. Interviewees suggested that this requires continued support that “must be integrated 
with professional development on instructional strategies in academic content.” While standalone 
SEL programs can be helpful resources for educators, they sometimes contribute to the sense that 
SEL is separate from the academic work that schools do. At the district level, the false division of 
“academics” and “student supports” (which often exist as separate departments within the district) 
creates a structural barrier to integration of these two related areas. 


Connecting Data to Instruction: Interviewees felt that the CORE Districts had done a great job of shat- 
ing information about the SQU, “especially the 40% that’s non-academic,” but indicated that more 
training and support could help connect the data to research and then to instructional strategies. For 
example, one principal said, “I have been using SE data from the district in our staff meetings to 
discuss what interventions would be helpful for students and to plan tier 1 and tier 2 interventions.” 
Annual assessments and school-level data will likely be insufficient for improving practice. A sup- 
plemental, formative approach to measurement could include assessing students’ SE skills regularly 
throughout the school year in a way that positively reinforces students who demonstrate social-emo- 
tional growth and provides more granular data for teachers to use in refining their practice over time. 


Over the next several months, the CORE Districts will continue to explore and iterate on next steps in each 
of these various areas, as well as additional areas of interest or need that emerge. 
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V. Key Lessons from the CORE Districts’ Approach 


Reflecting on the CORE Districts' approach, there are several takeaways that can inform other states’ and 
districts’ efforts to use innovative measures at scale as part of an expanded definition of student success. 


Explicitly articulate the key principles and non-negotiables up front. 


For the CORE Districts, these principles were college and career ready expectations for all students; a focus 
on collective responsibility, accountability, and action that emphasizes capacity building over high-stakes ac- 
countability; the development of intrinsic motivation for change through differentiated recognition, account- 
ability, and support for schools; and focused capacity-building for effective instruction and leadership. These 
were the guideposts that informed myriad decisions about implementation, the bedrock when challenges 
arose, and a clear signal to multiple stakeholders about the system’s priorities. 


District buy-in is critical when implementing an ambitious system. 


The CORE districts’ endorsement of the SQIS was crucial to its successful implementation. District buy-in 
stemmed from the fact that district leaders co-designed and opted into a novel approach to accountability 
and continuous improvement that reflected the districts’ own beliefs and values. They also made joint deci- 
sions about indicators and system implementation through participation on the CORE Districts' Board of 
Directors and reaffirmed their commitment to the approach by signing on to waiver amendments to USED. 
District staff also had many opportunities to share input and feedback during the waiver implementation and 
to collaborate with each other in ways that supported professional growth as well as buy-in to the SQIS. 


Prepare to iterate: No system of accountability and continuous improvement is perfect at the outset. 


When designing complex accountability and continuous improvement systems, it is important to be clear 
that iterations to the model will happen over time based on trends that emerge from the data, feedback from 
stakeholders, and developments in research. Throughout the process of iterating, the key principles of the 
system must remain constant, providing educators and community members with a sense of coherent system 
design. This was evident in the CORE Districts' waiver amendments: changes were requested to individual 
components of the waiver (e.g., to the method of calculating different indicators), but the CORE Districts’ 
key principles remained consistent. 


A multi-stage roll out can make novel measurement approaches more manageable. 


The CORE Districts tested and rolled out their new accountability system in several phases over the course 
of three years, making many changes along the way based on feedback from the districts. This process in- 
creased buy-in by giving the districts time to understand how each measure works and to participate in field 
testing before new measures were incorporated into the Index. Additionally, the phased rollout process al- 
lowed time for district staff to build cross-district relationships, connect with others who were wrestling with 
similar challenges, and develop new approaches for acting on the data once the system was fully implemented. 


The work requires support from multiple partners. 


In implementing the SQIS, the CORE Districts had support from a large number of different researchers, 
nonprofit organizations, funders, and stakeholders. These partners helped with a range of tasks: supporting 
the selection of SE and CC competencies, validating the SE measures, analyzing the SQII data, developing 
data reports for schools, providing professional development to participating districts, and evaluating the 
impact of the CORE Districts' system. Ultimately, the CORE Districts were able to harness the skills and 
perspectives of many partners to build a robust system that no single district would have been able to build 
alone. 
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Conclusion 


There is growing interest from education leaders and policymakers in understanding the CORE Districts’ 
SQIS because it provides an example of how we can think more expansively about the factors that support 
student success, which ESSA now enables all states to do. The CORE Districts’ ability to implement a holistic 
system of accountability and continuous improvement has been possible because the participating districts 
chose to hold themselves to high standards on a broad range of indicators that they believe are crucial to all 
students’ success. Together, the CORE districts have upheld a deep commitment to equity and a focus on 
using accountability as a “flashlight, not a hammer” choosing to “shine a brighter, wider light on the needs 

of all students and on their own educational strategies and practices” while constantly refining their own 
approach along the way.” In doing so, the CORE Districts provide a compelling example of what is possible 
when educators work together to create a system that reflects their deepest values and their firm commitment 
to preparing all students for college, career, and life. 


For states and districts that are interested in learning more specifics about how the CORE Districts have implemented compo- 
nents of their approach, please see the resources attached to this case study. 
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Appendix A: the CORE Districts’ Partners 


Center for Education Policy Research, Harvard University 


Education Analytics, Inc. 
Evelyn and Walter Haas, Jr. Fund 


Bill & Melinda Gates Foundation 


John W. Gardner Center for Youth and Their Communities, Stanford University 
The Dick and Charlene Kabcenell Family Foundation 


Panorama Education 


Policy Analysis for California Education (PACE), Stanford University, University of 


California—Davis, and the University of Southern California 


S.D. Bechtel, [R. Foundation, Stephen Bechtel Fund 


Silver Giving Foundation 


Stanford Center for Opportunity Policy in Education (SCOPE), Stanford University 


Stuart Foundation 


Transforming Education 


William and Flora Hewlett Foundation 


Versifit Technologies 
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Appendix B: The CORE Districts Timeline 


The CORE Districts implemented the Index and its accompanying new assessments in a sequential manner 
over the course of three years. This timeline highlights the key activities in the academic and SE/CC domains. 
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School Year 2013-2014 
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*  Social-emotional (SE) surveys piloted with ~9,000 students 
*  Culture-climate (CC) surveys in continued use via California Healthy Kids Survey process 


School Year 2014-2015 
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. SE and CC measures field tested as a unified survey with ~450,000 students and ~70,000 teachers 


School Year 2015-2016 


Man) odCcdanrevarecditeya 


. Full Index implementation, including academic performance/growth & SE/CC surveys administered* 

* Index reports released for CORE districts, excluding academic growth and SE/CC survey results 

* Second administration of social-emotional/culture-climate surveys to all students grades 4-12 in CORE 
districts 


School Year 2016-2017 


Implementation 


0 Index reports will include all indicators 


*Because California field-tested the SBAC assessment in 2013-2014 and did not have a statewide assessment, there was no 
baseline data to use to calculate growth measures in 2015. The CORE Districts will be able to release growth measures in fall 
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Appendix C: The CORE Districts’ School Identification Criteria 


Given the recent passage of The Every Student Succeeds Act (ESSA) and the expiration of No Child Left 
Behind (NCLB) waivers on August 1, 2016, the school identification provisions will no longer remain in place. 
However, it is instructive to explore how the CORE Districts designed their system as a model that other 
states may want to replicate when designing their school intervention systems pursuant to ESSA. 


PRIORITY: GRADUATION 


Bottom 10% on SQII for graduation 
rates; AND 

Graduation rate less than 60% in 
that year; AND 

Graduation rate less than 60% in 


the last 2 out of 3 years. 


PRIORITY: BOTTOM FIVE 
PERCENTILE IN ELA/MATH 


Bottom 10% on SQII; AND 
isYo)iColanmo)iam el=1ker-1al (om c= la) dine mia 
ELA or Math that year; AND 
sfo)ivo)aamoliam o\-1ce-lalti(-meslal dare mia 
ELA or Math in the last 2 out of 3 
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FOCUS: LOW ACHIEVING SUBGROUPS 


1. Bottom 10% on the SQII; AND 

2. Level 1 in subgroup performance 
on ELA, Math, or graduation that 
year; AND 
Lack of progress for the 
JU] ofe] gel0] 0 =) elma aal=ita(el >) mola Zalcera| 
there is concern 


FOCUS: WITHIN SCHOOL GAPS 


1. Lowest 5% in achievement gaps 
(over the last three years) 


FOCUS: LOW GRADUATION 


1. Graduation rate less than 60% that 
year; 

2. Graduation rate less than 60% in 
the last 2 out of 3 years. 


1. “CORE Districts Introduce New School Quality Improvement Index,” CORE Districts. Press release, December 4, 2015, 
http://coredistricts.org/wp-content/uploads/2015/12/Updated_CORE-Rollout-News-Release.pdf 


2. CORE Districts, “School Quality Improvement Index,” p. 1, http://coredistricts.org/wp-content/uploads/2015/12/CORE 
informationsheet_SQII.pdf 


3. “CORE Districts Introduce New School Quality Improvement Index,” CORE Districts. Press release, December 4, 2015, 
http://coredistricts.org/wp-content/uploads/2015/12/Updated_CORE-Rollout-News-Release.pdf 


4. These districts that received the waiver were Fresno, Long Beach, Los Angeles, Oakland, San Francisco, and Santa 
Ana. 


5. For amore detailed history of how the CORE Districts came together, see Joel Knudson and Mark Garibaldi, “None of 
Us Are as Good As All of Us: Early Lessons from the CORE Districts,” American Institutes for Research, August 2015, http:// 
coredistricts.org/wp-content/uploads/2015/08/CORE-Cross-District-Collaboration-Report-August-2015.pdf. See also Califor- 
nia Collaborative on District Reform, http://www.cacollaborative.org/ and “Public/Private Partnership Celebrates 15 Years of 
Dialogue,” Price Philanthropies, August 19, 2014, http://pricephilanthropies.org/tag/urban-education-dialogue/ 


6. Ibid. 
7. Ibid. 


8. U.S. Department of Education, “ESEA Flexibility,” http://www2.ed.gov/policy/elsec/quid/esea-flexibility/index.html 


9. They worked in conjunction with the Parthenon Group, now Parthenon-EY. 


10. For more information on the longitudinal research base showing the impact of social-emotional competencies on stu- 
dents’ outcomes, see Chris Gabrieli, Dana Ansel, and Sara Bartolino Krachman, “Ready to Be Counted: The Research Case 
for Education Policy Action on Non-Cognitive Skills,” December 2015, http://www.transformingeducation.org/s/ReadytoBe- 
Counted_Release.pdf 


11. Oakland and Sacramento, two of the CORE districts, were part of the Collaborative for Academic, Social, and Emo- 
tional Learning (CASEL)’s Collaborating Districts Initiative, which aims to develop students’ social-emotional skills through 
district-level systemic change. See http://www.casel.org/collaborating-districts/ 


12. Cover letter for ESEA Flexibility Request for Window 3 (CORE Districts waiver application), February 28, 2013 http:// 
www2.ed.gov/nclb/freedom/local/flexibility/waiverletters2009/cacoreflexrequest22013.pdf 


13. This language is from the CORE Districts waiver application at: http://Awww2.ed.gov/policy/eseaflex/approved-requests/ 
cacorerenewalreq92415.pdf 


14. The CORE Districts submitted a waiver amendment in 2015 that combined culture/climate factors (previously 20% of the 
index) with social-emotional skills (previously 20% of the index) into a single category worth 40% of the total index score. 
Several factors led to that decision: both components required the collection of survey data, and participating schools felt 
that the two components were closely related to one another. 


15. The CORE Districts waiver application, February 28, 2013, p.38, http://www2.ed.gov/policy/eseaflex/approved-requests/ 
cacorerenewalreq92415.pdf 


16. The CORE Districts use the terminology “meaningful, measurable, and actionable.” 
17. Carol Dweck, Mindset: The New Psychology of Success (New York: Random House, 2006). 
18. Albert Bandura, Self-Efficacy: The Exercise of Control (New York: W.H. Freeman and Company, 1997). 


19. Collaborative for Academic, Social, and Emotional Learning, “Social and Emotional Learning Core Competencies,” 
http://www.casel.org/social-and-emotional-learning/core-competencies 
20. Ibid. 


21. Smarter Balanced Assessment Consortium (SBAC) is a test designed to assess students’ performance and mastery of 
the Common Core State Standards. For more information, see http://www.smarterbalanced.org/ 


22. These required a waiver amendment. 
23. These required a waiver amendment. 


24. For background about how the CORE Districts determined this measure, see Antwan Wilson, “How to Assess Whether 
Middel Schoolers Are Ready for High School,” EdSource, September 15, 2015, http://edsource.org/2015/how-to-assess- 
whether-middle-schoolers-are-ready-for-high-school/86094 

25. Martin R. West, “Should Non-Cognitive Skills be Included in School Accountability Systems? Preliminary Evidence from 


California’s CORE Districts,” Brookings Institution, March 17, 2016, http:/Awww.brookings.edu/research/reports/2016/03/17- 
non-cognitive-skills-school-accountability-california-core-west 
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26. See, e.g., Martin R. West, et al., “Promise and Paradox: Measuring Students’ Non-Cognitive Skills and the Impact of 
Schooling,” Educational Evaluation and Policy Analysis, August 2015. The paper explores this paradoxical effect based on 
research that Transforming Education and its partners conducted in Boston schools. 


27. Ibid. 


28. See, e.g., Philip M. Podsakoff, Scott MacKenzie, and Jeong-Yeon Lee, “Common Method Biases in Behavioral Re- 
search: A Critical Review of the Literature and Recommended Remedies.” Journal of Applied Psychology 88, no.5 (2003): 
879-903. 


29. See, e.g., Steven J. Spencer, Claude M. Steele, and Diane M. Quinn, “Stereotype Threat and Women’s Math Perfor- 
mance.” Journal of Experimental Social Psychology 35, no. 1 (1999): 4—28. 


30. Ibid. 


31. Internal reliability is a measure of the degree to which items included in the test assess the same underlying construct. 
The general rule of thumb is that assessments must have reliability coefficients above 0.7 to be considered reliable. 


32. See Martin R. West, “Should Non-Cognitive Skills be Included in School Accountability Systems? Preliminary Evi- 
dence from California’s CORE Districts,” Brookings Institution, March 17, 2016, http://www.brookings.edu/research/re- 
ports/2016/03/17-non-cognitive-skills-school-accountability-california-core-west 


33. All of the correlations reported were statistically significant at the .001 level, with the exception of the correlation be- 
tween growth mindset and suspensions, which was significant at the .01 level. 


34. Analyses conducted by Harvard University’s Center for Education Policy Research (CEPR) for the CORE Districts and 
Transforming Education. 


35. Analyses conducted by Harvard's CEPR for the CORE Districts and Transforming Education. Teacher Report (Student 
SE Skills Overall) is a composite of teacher-rated self-management and teacher-rated social awareness (n = 71,060). 


36. Angela L. Duckworth and David Scott Yeager, “Measurement Matters: Assessing Personal Qualities Other Than Cogni- 
tive Ability for Educational Purposes.” Educational Researcher 44, no. 4 (May 13, 2015): 237-251. 


37. For more information, see Martin R. West, “Should Non-Cognitive Skills be Included in School Accountability Systems? 
Preliminary Evidence from California’s CORE Districts,” Brookings Institution, March 17, 2016, htto://www.brookings.edu/ 


research/reports/2016/03/17-non-cognitive-skills-school-accountability-california-core-west 

38. See Sara Bartolino Krachman, “Let’s Not Let Perfect Be the Enemy of Good: Measurement, MESH, and Continuous 
Improvement.” Transforming Education, March 2, 2016, http://www.transformingeducation.org/blog/2016/3/2/lets-not-let-per- 
fect-be-the-enemy-of-good-measurement-mesh-and-continuous-improvement 


39. A forthcoming policy paper to be released by TransformEd will explore the policy implications of these measures in 
greater depth. 


40. For these items developed by WestEd and CDE, see http://coredistricts.org/wp-content/uploads/2015/10/SE-CC-Do- 
main-School-Culture-Climate-Surveys-updated-2.18.15.pdf 


41. Ibid. 


42. Amrit Thapa et al., “A Review of School Climate Research.” Review of Educational Research 83 (September 2013): 
365-366, http://rer.sagepub.com/content/83/3/357.full.pdf+html. 
43. See Amrit Thapa, et al., p.363-364. See also Higgins-D’Alessandro, A., & Sakwarawich, A. (2011, October). “Congru- 


ency and Determinants of Teacher and Student Views of School Culture.” Paper presented at the Association for Moral 
Education Annual Conference, Nanjing, China. Reported in Review of Educational Research 83, p.364. 


44. Anthony S. Bryk, et al., Organizing Schools for Improvement: Lessons from Chicago (Chicago: University of Chicago 
Press, 2010). 


45. These data reports are courtesy of Panorama Education. 
46. See, e.g., Michael Fullan, “Choosing the Wrong Drivers for Whole System Reform,” Center for Strategic Education, May 
2011, http://michaelfullan.ca/wp-content/uploads/2016/06/13501655630.pdf 


47. Per the California Education Code, School Site Councils are required to: a.) “Measure effectiveness of improvement 
strategies at the school. b.) Seek input from school advisory committees. c.) Reaffirm or revise school goals. d.) Revise 
improvement strategies and expenditures. e.) Recommend the approved single plan for student achievement (SPSA) to 
the governing board. f.) Monitor implementation of the SPSA.” For more information, see http://pubs.cde.ca.gov/tcsii/ch9/ 


sscldrshp.aspx 
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48. The CORE Districts’ “School Quality Improvement System School Communities of Practice Program Feedback” (July 
2015); number of respondents: 30. 


49. Reward schools are those with high performance or high growth, and for each of these categories there is an absence 
of stagnant achievement gaps. Additionally, focus schools have the option of being part of a school pairing, rather than a 
community of practice, but only two schools chose to do so. 


50. The CORE Districts' “School Quality Improvement System School Pairing Program Feedback” (July 2015); number of 
respondents: 27 reward schools; 20 focus/priority schools. 


51. SIG funds are given to state education agencies for competitive grants to school districts that have low-performing 
schools. For more information about SIG, see http://www2.ed.gov/programs/sif/index.html| 


52. For more information, see http://coredistricts.org/wp-content/uploads/2015/12/CORE-informationsheet.pdf 


53. PACE is a research center that draws from Stanford University, the University of Southern California, and the University 
of California (Davis). 


54. TransformEd and the CORE Districts recently conducted a limited set of semi-structured interviews with CORE district 
leaders, principals, and teachers to get a sense of how educators are making sense of the data on SE skills and where they 
need more support. Initial findings from these interviews are included here and will be refined further through additional 
focus groups and interviews. 


55. Michael Hanson, Superintendent of the Fresno Unified School District, quoted in “School Quality Improvement Index,” p. 
1, http://coredistricts.org/wp-content/uploads/2015/12/CORE-informationsheet.pdf 
56. These definitions of school categories were slated to apply to the 2016-2017 school year. However, given the recent 


passage of ESSA and the expiration of waivers on August 1, 2016, these provisions will not be in place for 2016-2017. See 
https://www2.ed.gov/policy/eseaflex/approved-requests/cacorerenewalreq92415.pdf (pp. 121-124). 


57. These definitions of school categories were slated to apply to the 2016-2017 school year. However, given the recent 
passage of ESSA and the expiration of waivers on August 1, 2016, these provisions will not be in place for 2016-2017. See 


https://www2.ed.gov/policy/eseaflex/approved-requests/cacorerenewalreq92415.pdf (pp.140-—142). 
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